gremlin/Getty Images
- Schools often use AI to detect students at risk of self-harm, violence against others, and bullying.
- Some worry these programs violate students' privacy and freedom to use the internet, WSJ reports.
- One AI company said it detected 5,000 self-harm or suicide situations among students in a single week.
- See more stories on Insider's business page.
As students head back to school, school districts across the country are using artificial-intelligence programs to monitor their communications and web searches to detect those at risk of self-harm, bullying, violence, and other mental health issues, a report from the Center for Democracy and Technology said.
While surveillance programs in schools are widespread – 71% of teachers in the CDT survey said their schools used monitoring software – critics say the use of these programs can infringe on students' privacy and can lead to a slippery slope if schools begin to use the AI for reasons like discipline, instead of harm prevention.
The software typically scans for certain key words and phrases on school-issued devices and alerts an administrator when it finds a targeted key phrase, The Wall Street Journal reported. The administrator can determine if the alert is credible and choose to notify parents or intervene with the student, according to the Journal.
Bark, a monitoring company used by 2,900 school districts, told the Journal it detected 5,000 self-harm or suicide situations in a single week. The San Marcos Unified School District, which uses the monitoring software Lightspeed, told the Journal that it recieved five credible alerts about self-harm during the first week of school alone.
In one case, the software notified the school about a student who wrote that they wanted to die in a diary kept on a Google Doc. Campbell said the school notified the child's parents, who then got help for their child.
According to a survey by The Harris Poll, children's mental health has declined throughout the pandemic, with parents of school-age children reporting increased rates of constant worrying, frequent sadness, and nervous habits among their children.
Though monitoring programs attract criticism from privacy and security experts, 66% of teachers agreed that the benefits of online monitoring outweighed the risks, and 50% of students reported they were comfortable with the use of monitoring software, according to the CDT survey.
However, Chad Marlow, a privacy expert at the American Civil Liberties Union, told The Guardian in 2019 that there isn't any evidence to suggest that surveilling students will make them safer. While many student surveillance companies in the $3 billion-a-year school security industry promote their programs through anecdotal evidence and self-collected data, there hasn't been an independent analysis of whether surveillance actually reduces rates of violence or self-harm among students.
The widespread awareness of monitoring software has already changed student behavior and made it more difficult to spot the harmful behavior the monitoring programs try to detect. Six in ten students said they don't share their true thoughts online since they know they're being monitored, and 80% said they were more careful about their online searches because they know their activity is monitored, the CDT report said.
Around 60% of teachers and parents in the CDT survey said that monitoring software could cause long-term harm if information collected was used for discipline or shared out of context, and around 50% said monitoring could have unintended negative consequences for LGBTQ students.
According to a study by the Brennan Center for Justice, schools spent $1 million in 2018 on monitoring students' social media. Social-media monitoring can be particularly dangerous for students of color, who are often more likely to be disproportionately punished by schools, the study noted.
"Overall, research shows that as school security measures proliferate, students often feel less safe," the Brennan Center report said. "And overbroad and unnecessary surveillance is likely to have a detrimental impact on students' privacy and chill their ability to express themselves."